Toward a theory of perspective perception in pictures.

This paper reviews projection models and their perception in realistic pictures, and proposes hypotheses for three-dimensional (3D) shape and space perception in pictures. In these hypotheses, eye fixations, and foveal vision play a central role. Many past theories and experimental studies focus solely on linear perspective. Yet, these theories fail to explain many important perceptual phenomena, including the effectiveness of nonlinear projections. Indeed, few classical paintings strictly obey linear perspective, nor do the best distortion-avoidance techniques for wide-angle computational photography. The hypotheses here employ a two-stage model for 3D human vision. When viewing a picture, the first stage perceives 3D shape for the current gaze. Each fixation has its own perspective projection, but, owing to the nature of foveal and peripheral vision, shape information is obtained primarily for a small region of the picture around the fixation. As a viewer moves their eyes, the second stage continually integrates some of the per-gaze information into an overall interpretation of a picture. The interpretation need not be geometrically stable or consistent over time. It is argued that this framework could explain many disparate pictorial phenomena, including different projection styles throughout art history and computational photography, while being consistent with the constraints of human 3D vision. The paper reviews open questions and suggests new studies to explore these hypotheses.

Subicular neurons encode concave and convex geometries.

Animals in the natural world constantly encounter geometrically complex landscapes. Successful navigation requires that they understand geometric features of these landscapes, including boundaries, landmarks, corners and curved areas, all of which collectively define the geometry of the environment1-12. Crucial to the reconstruction of the geometric layout of natural environments are concave and convex features, such as corners and protrusions. However, the neural substrates that could underlie the perception of concavity and convexity in the environment remain elusive. Here we show that the dorsal subiculum contains neurons that encode corners across environmental geometries in an allocentric reference frame. Using longitudinal calcium imaging in freely behaving mice, we find that corner cells tune their activity to reflect the geometric properties of corners, including corner angles, wall height and the degree of wall intersection. A separate population of subicular neurons encode convex corners of both larger environments and discrete objects. Both corner cells are non-overlapping with the population of subicular neurons that encode environmental boundaries. Furthermore, corner cells that encode concave or convex corners generalize their activity such that they respond, respectively, to concave or convex curvatures within an environment. Together, our findings suggest that the subiculum contains the geometric information needed to reconstruct the shape and layout of naturalistic spatial environments.

Representation of Natural Contours by a Neural Population in Monkey V4.

The cortical visual area, V4, has been considered to code contours that contribute to the intermediate-level representation of objects. The neural responses to the complex contour features intrinsic to natural contours are expected to clarify the essence of the representation. To approach the cortical coding of natural contours, we investigated the simultaneous coding of multiple contour features in monkey (Macaca fuscata) V4 neurons and their population-level representation. A substantial number of neurons showed significant tuning for two or more features such as curvature and closure, indicating that a substantial number of V4 neurons simultaneously code multiple contour features. A large portion of the neurons responded vigorously to acutely curved contours that surrounded the center of classical receptive field, suggesting that V4 neurons tend to code prominent features of object contours. The analysis of mutual information (MI) between the neural responses and each contour feature showed that most neurons exhibited similar magnitudes for each type of MI, indicating that many neurons showing the responses depended on multiple contour features. We next examined the population-level representation by using multidimensional scaling analysis. The neural preferences to the multiple contour features and that to natural stimuli compared with silhouette stimuli increased along with the primary and secondary axes, respectively, indicating the contribution of the multiple contour features and surface textures in the population responses. Our analyses suggested that V4 neurons simultaneously code multiple contour features in natural images and represent contour and surface properties in population.

Perception of 3D shape integrates intuitive physics and analysis-by-synthesis.

Many surface cues support three-dimensional shape perception, but humans can sometimes still see shape when these features are missing-such as when an object is covered with a draped cloth. Here we propose a framework for three-dimensional shape perception that explains perception in both typical and atypical cases as analysis-by-synthesis, or inference in a generative model of image formation. The model integrates intuitive physics to explain how shape can be inferred from the deformations it causes to other objects, as in cloth draping. Behavioural and computational studies comparing this account with several alternatives show that it best matches human observers (total n = 174) in both accuracy and response times, and is the only model that correlates significantly with human performance on difficult discriminations. We suggest that bottom-up deep neural network models are not fully adequate accounts of human shape perception, and point to how machine vision systems might achieve more human-like robustness.

Modeling the Role of Contour Integration in Visual Inference.

Under difficult viewing conditions, the brain's visual system uses a variety of recurrent modulatory mechanisms to augment feedforward processing. One resulting phenomenon is contour integration, which occurs in the primary visual (V1) cortex and strengthens neural responses to edges if they belong to a larger smooth contour. Computational models have contributed to an understanding of the circuit mechanisms of contour integration, but less is known about its role in visual perception. To address this gap, we embedded a biologically grounded model of contour integration in a task-driven artificial neural network and trained it using a gradient-descent variant. We used this model to explore how brain-like contour integration may be optimized for high-level visual objectives as well as its potential roles in perception. When the model was trained to detect contours in a background of random edges, a task commonly used to examine contour integration in the brain, it closely mirrored the brain in terms of behavior, neural responses, and lateral connection patterns. When trained on natural images, the model enhanced weaker contours and distinguished whether two points lay on the same versus different contours. The model learned robust features that generalized well to out-of-training-distribution stimuli. Surprisingly, and in contrast with the synthetic task, a parameter-matched control network without recurrence performed the same as or better than the model on the natural-image tasks. Thus, a contour integration mechanism is not essential to perform these more naturalistic contour-related tasks. Finally, the best performance in all tasks was achieved by a modified contour integration model that did not distinguish between excitatory and inhibitory neurons.

Gestalt neurons and emergent properties in visual perception: A novel concept for the transformation from local to global processing.

Gestalten in visual perception are defined by emergent properties of the whole, which cannot be predicted from the sum of its parts; rather, they arise by virtue of inherent principles, the Laws of Seeing. This review attempts to assign neurophysiological correlates to select emergent properties in motion and contour perception and proposes parallels to the processing of local versus global attributes by classical versus contextual receptive fields. The aim is to identify Gestalt neurons in the visual system to account for the Laws of Seeing in causal terms and to explain "Why do things look as they do" (Koffka, 1935, p. 76).

Planning to revisit: Neural activity in refixation precursors.

Eye tracking studies suggest that refixations-fixations to locations previously visited-serve to recover information lost or missed during earlier exploration of a visual scene. These studies have largely ignored the role of precursor fixations-previous fixations on locations the eyes return to later. We consider the possibility that preparations to return later are already made during precursor fixations. This process would mark precursor fixations as a special category of fixations, that is, distinct in neural activity from other fixation categories such as refixations and fixations to locations visited only once. To capture the neural signals associated with fixation categories, we analyzed electroencephalograms (EEGs) and eye movements recorded simultaneously in a free-viewing contour search task. We developed a methodological pipeline involving regression-based deconvolution modeling, allowing our analyses to account for overlapping EEG responses owing to the saccade sequence and other oculomotor covariates. We found that precursor fixations were preceded by the largest saccades among the fixation categories. Independent of the effect of saccade length, EEG amplitude was enhanced in precursor fixations compared with the other fixation categories 200 to 400 ms after fixation onsets, most noticeably over the occipital areas. We concluded that precursor fixations play a pivotal role in visual perception, marking the continuous occurrence of transitions between exploratory and exploitative modes of eye movement in natural viewing behavior.

The Emergence of Tuning to Global Shape Properties of Radial Frequency Patterns in the Ventral Visual Pathway.

Radial frequency (RF) patterns, created by sinusoidal modulations of a circle's radius, are processed globally when RF is low. These closed shapes therefore offer a useful way to interrogate the human visual system for global processing of curvature. RF patterns elicit greater responses than those to radial gratings in V4 and more anterior face-selective regions of the ventral visual pathway. This is largely consistent with work on nonhuman primates showing curvature processing emerges in V4, but is evident also higher up the ventral visual stream. Rather than contrasting RF patterns with other stimuli, we presented them at varied frequencies in a regimen that allowed tunings to RF to be derived from 8 human participants (3 female). We found tuning to low RF in lateral occipital areas and to some extent in V4. In a control experiment, we added a high-frequency ripple to the stimuli disrupting the local contour. Low-frequency tuning to these stimuli remained in the ventral visual stream, underscoring its role in global processing of shape curvature. We then used representational similarity analysis to show that, in lateral occipital areas, the neural representation was related to stimulus similarity, when it was computed with a model that captured how stimuli are perceived. We therefore show that global processing of shape curvature emerges in the ventral visual stream as early as V4, but is found more strongly in lateral occipital regions, which exhibit responses and representations that relate well to perception.SIGNIFICANCE STATEMENT We show that tuning to low radial frequencies, known to engage global shape processing mechanisms, was localized to lateral occipital regions. When low-level stimulus properties were accounted for such tuning emerged in V4 and LO2 in addition to the object-selective region LO. We also documented representations of global shape properties in lateral occipital regions, and these representations were predicted well by a proxy of the perceptual difference between the stimuli.

Qualitative perception of 3D shape from patterns of luminance curvature.

A new source of information is proposed for the perception of three-dimensional (3D) shape from shading that identifies surface concavities from the curvature of the luminance field. Two experiments measured the abilities of human observers to identify concavities on smoothly curved shaded surfaces depicted with several different patterns of illumination and several different material properties. Observers were required to identify any apparent concavities along designated cross sections of the depicted objects and to mark each concavity with an adjustable dot. To analyze the results, we computed both the surface curvature and the luminance curvature along each image cross section. The results revealed that most responses were in concave regions of the luminance profiles, although they were often shifted in phase relative to the curvature of the depicted surfaces. This pattern of performance was surprisingly robust over large changes in the pattern of illumination or surface material properties. Our analysis predicts that observers should make false alarm responses in regions where a luminance concavity does not correspond to a surface concavity, and our empirical results confirm that prediction.

Learning a Model of Shape Selectivity in V4 Cells Reveals Shape Encoding Mechanisms in the Brain.

The mechanisms involved in transforming early visual signals to curvature representations in V4 are unknown. We propose a hierarchical model that reveals V1/V2 encodings that are essential components for this transformation to the reported curvature representations in V4. Then, by relaxing the often-imposed prior of a single Gaussian, V4 shape selectivity is learned in the last layer of the hierarchy from Macaque V4 responses. We found that V4 cells integrate multiple shape parts from the full spatial extent of their receptive fields with similar excitatory and inhibitory contributions. Our results uncover new details in existing data about shape selectivity in V4 neurons that with additional experiments can enhance our understanding of processing in this area. Accordingly, we propose designs for a stimulus set that allow removing shape parts without disturbing the curvature signal to isolate part contributions to V4 responses.SIGNIFICANCE STATEMENT Selectivity to convex and concave shape parts in V4 neurons has been repeatedly reported. Nonetheless, the mechanisms that yield such selectivities in the ventral stream remain unknown. We propose a hierarchical computational model that incorporates findings of the various visual areas involved in shape processing and suggest mechanisms that transform the shape signal from low-level features to convex/concave part representations. Learning shape selectivity from Macaque V4 responses in the final processing stage in our model, we found that V4 neurons integrate shape parts from the full spatial extent of their receptive field with both facilitatory and inhibitory contributions. These results reveal hidden information in existing V4 data that with additional experiments can enhance our understanding of processing in V4.

Beauty and the sharp fangs of the beast: degree of angularity predicts perceived preference and threat.

The evaluation of angular vs. curved forms has a long history in psychology but few of the many studies conducted have examined actual degree of angularity. In two experiments, we present observers with randomly positioned and randomly oriented texture displays of angles viewed within a circular frame. The angle conditions varied from 0° to 180° in 20° increments, covering the entire spectrum of possibilities including acute, obtuse, right, and straight line angles. In Experiment 1, 25 undergraduates rated the perceived beauty of these displays. In Experiment 2, the same stimulus set and procedure were used with 27 participants instead judging perceived threat. Based on the findings in the literature, we predicted that sharper angles would be judged less beautiful and more threatening. The results were mostly confirmed. Acute angles are preferred less but there are also distinct preferences for right angles and straight lines, perhaps due to their greater familiarity in constructed environments. There was a consistent and anticipated finding for threat in the second study: the sharper an angle, the greater its perceived threat. Fear of sharp objects as assessed in a personality questionnaire was found to positively correlate with threat judgements. Future work should look more closely at degree of angularity in embedded object contours and at individual response differences.

Object Image Size Is a Fundamental Coding Dimension in Human Vision: New Insights and Model.

In previous psychophysical work we found that luminance contrast is integrated over retinal area subject to contrast gain control. If different mechanisms perform this operation for a range of superimposed retinal regions of different sizes, this could provide the basis for size-coding. To test this idea we included two novel features in a standard adaptation paradigm to discount more pedestrian accounts of repulsive size-aftereffects. First, we used spatially jittering luminance-contrast adaptors to avoid simple contour displacement aftereffects. Second, we decoupled adaptor and target spatial frequency to avoid the well-known spatial frequency shift aftereffect. Empirical results indicated strong evidence of a bidirectional size adaptation aftereffect. We show that the textbook population model is inappropriate for our results, and develop our existing model of contrast perception to include multiple size mechanisms with divisive surround-suppression from the largest mechanism. For a given stimulus patch, this delivers a blurred step-function of responses across the population, with contrast and size encoded by the height and lateral position of the step. Unlike for textbook population coding schemes, our human results (N = 4 male, N = 4 female) displayed two asymmetries: (i) size aftereffects were greatest for targets smaller than the adaptor, and (ii) on that side of the function, results did not return to baseline, even when targets were 25% of adaptor diameter. Our results and emergent model properties provide evidence for a novel dimension of visual coding (size) and a novel strategy for that coding, consistent with previous results on contrast detection and discrimination for various stimulus sizes.

Tilt aftereffect spreads across the visual field.

The tilt aftereffect (TAE) is observed when adaptation to a tilted contour alters the perceived tilt of a subsequently presented contour. Thus far, TAE has been treated as a local aftereffect observed only at the location of the adapter. Whether and how TAE spreads to other locations in the visual field has not been systematically studied. Here, we sought an answer to this question by measuring TAE magnitudes at locations including but not limited to the adapter location. The adapter was a tilted grating presented at the same peripheral location throughout an experimental session. In a single trial, participants indicated the perceived tilt of a test grating presented after the adapter at one of fifteen locations in the same visual hemifield as the adapter. We found non-zero TAE magnitudes in all locations tested, showing that the effect spreads across the tested visual hemifield. Next, to establish a link between neuronal activity and behavioral results and to predict the possible neuronal origins of the spread, we built a computational model based on known characteristics of the visual cortex. The simulation results showed that the model could successfully capture the pattern of the behavioral results. Furthermore, the pattern of the optimized receptive field sizes suggests that mid-level visual areas, such as V4, could be critically involved in TAE and its spread across the visual field.

BOLD activation on the groundside of figures: More suppression of grounds that competed more for figural status.

A fundamental aspect of object detection is assigning a border to one (figure) side but not the other (ground) side. Figures are shaped; grounds appear shapeless near the figure border. Accumulating evidence supports the view that the mechanism of figure assignment is inhibitory competition with the figure perceived on the winning side. Suppression has been observed on the groundside of figure borders. One prediction is that more suppression will be observed when the groundside competes more for figural status. We tested this prediction by assessing BOLD activation on the groundside of two types of stimuli with articulated borders: AEnov and AEfam stimuli. In both stimulus types, multiple image-based priors (symmetry, closure, small area, enclosure by a larger region) favored the inside as the figure. In AEfam but not AEnov stimuli, the figural prior of familiar configuration present on the outside competes for figural status. Observers perceived the insides of both types of stimuli as novel figures and the outsides as shapeless grounds. Previously, we observed lower BOLD activation in early visual areas representing the grounds of AEfam than AEnov stimuli, although unexpectedly, activation was above baseline. With articulated borders, it can be difficult to exclude figure activation from ground ROIs. Here, our ground ROIs better excluded figure activation; we also added straight-edge (SE) control stimuli and increased the sample size. In early visual areas representing the grounds, we observed lower BOLD activation on the groundside of AEfam than AEnov stimuli and below-baseline BOLD activation on the groundside of SE and AEfam stimuli. These results, indicating that greater suppression is applied to groundsides that competed more for figural status but lost the competition, support a Bayesian model of figure assignment in which proto-objects activated at both low and high levels where image features and familiar configurations are represented, respectively, compete for figural status.

Horizontal connectivity in V1: Prediction of coherence in contour and motion integration.

This study demonstrates the functional importance of the Surround context relayed laterally in V1 by the horizontal connectivity, in controlling the latency and the gain of the cortical response to the feedforward visual drive. We report here four main findings: 1) a centripetal apparent motion sequence results in a shortening of the spiking latency of V1 cells, when the orientation of the local inducer and the global motion axis are both co-aligned with the RF orientation preference; 2) this contextual effects grows with visual flow speed, peaking at 150-250°/s when it matches the propagation speed of horizontal connectivity (0.15-0.25 mm/ms); 3) For this speed range, the axial sensitivity of V1 cells is tilted by 90° to become co-aligned with the orientation preference axis; 4) the strength of modulation by the surround context correlates with the spatiotemporal coherence of the apparent motion flow. Our results suggest an internally-generated binding process, linking local (orientation /position) and global (motion/direction) features as early as V1. This long-range diffusion process constitutes a plausible substrate in V1 of the human psychophysical bias in speed estimation for collinear motion. Since it is demonstrated in the anesthetized cat, this novel form of contextual control of the cortical gain and phase is a built-in property in V1, whose expression does not require behavioral attention and top-down control from higher cortical areas. We propose that horizontal connectivity participates in the propagation of an internal "prediction" wave, shaped by visual experience, which links contour co-alignment and global axial motion at an apparent speed in the range of saccade-like eye movements.

The strength of feedback processing is associated with resistance to visual backward masking during Illusory Contour processing in adult humans.

Re-entrant feedback processing is a key mechanism of visual object-recognition, especially under compromised viewing conditions where only sparse information is available and object features must be interpolated. Illusory Contour stimuli are commonly used in conjunction with Visual Evoked Potentials (VEP) to study these filling-in processes, with characteristic modulation of the VEP in the ∼100-150 ms timeframe associated with this re-entrant processing. Substantial inter-individual variability in timing and amplitude of feedback-related VEP modulation is observed, raising the question whether this variability might underlie inter-individual differences in the ability to form strong perceptual gestalts. Backward masking paradig ms have been used to study inter-individual variance in the ability to form robust object perceptions before processing of the mask interferes with object-recognition. Some individuals recognize objects when the time between target object and mask is extremely short, whereas others struggle to do so even at longer target-to-mask intervals. We asked whether timing and amplitude of feedback-related VEP modulations were associated with individual differences in resistance to backward masking. Participants (N=40) showed substantial performance variability in detecting Illusory Contours at intermediate target-to-mask intervals (67 ms and 117 ms), allowing us to use kmeans clustering to divide the population into four performance groups (poor, low-average, high-average, superior). There was a clear relationship between the amplitude (but not the timing) of feedback-related VEP modulation and Illusory Contour detection during backward masking. We conclude that individual differences in the strength of feedback processing in neurotypical humans lead to differences in the ability to quickly establish perceptual awareness of incomplete visual objects.

Perceptual scale for transparency: Common fate overrides geometrical and color cues.

Objects that pass light through are considered transparent, and we generally expect that the light coming out will match the color of the object. However, when the object is placed on a colored surface, the light coming back to our eyes becomes a composite of surface, illumination, and transparency properties. Despite that, we can often perceive separate overlaid and overlaying layers differing in colors. How neurons separate the information to extract the transparent layer remains unknown, but the physical characteristics of transparent filters generate geometrical and color features in retinal images, which could provide cues for separating layers. We estimated the relative importance of such cues in a perceptual scale for transparency, using stimuli in which X- or T-junctions, different relative motions, and consistent or inconsistent colors cooperated or competed in forced-preference psychophysics experiments. Maximum-likelihood Thurstone scaling revealed that motion increased transparency for X-junctions, but decreased transparency for T-junctions by creating the percept of an opaque patch. However, if the motion of a filter uncovered a dynamically changing but stationary pattern, sharing a common fate with the surround but forming T-junctions, the probability of seeing transparency was almost as high as for moving X-junctions, despite the stimulus being physically improbable. In addition, geometric cues overrode color inconsistency to a great degree. Finally, a linear model of transparency perception as a function of relative motions between filter, overlay, and surround layers, contour continuation, and color consistency, quantified a hierarchy of latent influences on when the filter is seen as a separate transparent layer.

Increased insula activity precedes the formation of subjective illusory Gestalt.

The constructive nature of human perception sometimes leads us to perceiving rather complex impressions from simple sensory input: for example, recognizing animal contours in cloud formations or seeing living creatures in shadows of objects. A special type of bistable stimuli gives us a rare opportunity to study the neural mechanisms behind this process. Such stimuli can be visually interpreted either as simple or as more complex illusory content on the basis of the same sensory input. Previous studies demonstrated increased activity in the superior parietal cortex during the perception of an illusory Gestalt impression compared to a simpler interpretation. Here, we examined the role of slow fluctuations of resting-state fMRI activity in shaping the subsequent illusory interpretation by investigating activity related to the illusory Gestalt not only during, but also prior to its perception. We presented 31 participants with a bistable motion stimulus, which can be perceived either as four moving dot pairs (local) or two moving illusory squares (global). fMRI was used to measure brain activity in a slow event-related design. We observed stronger IPS and putamen responses to the stimulus when participants perceived the global interpretation compared to the local, confirming the findings of previous studies. Most importantly, we also observed that the global stimulus interpretation was preceded by an increased activity of the bilateral dorsal insula, which is known to process saliency and gate information for conscious access. Our data suggest an important role of the dorsal insula in shaping complex illusory interpretations of the sensory input.

The role of color contrast gain control in global form perception.

A Glass pattern consists of randomly distributed dot pairs, or dipoles, whose orientation is determined by a geometric transform, which defines the global percept perceived by an observer. The perception of Glass patterns involves a local process that associates dot pairs into dipoles and a global process that groups the dipoles into a global structure. In the present study, we used a variant of Glass patterns, which was composed of randomly distributed tripoles instead of dipoles, to estimate the influence of color contrast on perceptual grouping. Each tripole contained an anchor dot and two context dots. Grouping the anchor dot with one of the context dots resulted in a global percept of a clockwise spiral, while grouping with the other dot, a counter-clockwise spiral. All dots in each pattern were modulated in the same color direction but different contrasts. Four colors were involved, namely, red, green, blue, and yellow. The observers were to determine whether the spiral in each trial was clockwise or counter-clockwise. The probability of a context dot being grouped with the anchoring dot increased with its color contrast to a certain level, then decreased when the contrast continued to increase. Such probability decreased as the contrast of the other context dot increased. Our result cannot be explained by existing models in the literature, but with a divisive inhibition model. The equiluminance contrast result observed here is similar to the inverted U-shaped function for luminance contrast result previously reported by us, except that the color contrast model comprises a weaker self-inhibition component.

Visual discrimination of optical material properties: A large-scale study.

Complex visual processing involved in perceiving the object materials can be better elucidated by taking a variety of research approaches. Sharing stimulus and response data is an effective strategy to make the results of different studies directly comparable and can assist researchers with different backgrounds to jump into the field. Here, we constructed a database containing several sets of material images annotated with visual discrimination performance. We created the material images using physically based computer graphics techniques and conducted psychophysical experiments with them in both laboratory and crowdsourcing settings. The observer's task was to discriminate materials on one of six dimensions (gloss contrast, gloss distinctness of image, translucent vs. opaque, metal vs. plastic, metal vs. glass, and glossy vs. painted). The illumination consistency and object geometry were also varied. We used a nonverbal procedure (an oddity task) applicable for diverse use cases, such as cross-cultural, cross-species, clinical, or developmental studies. Results showed that the material discrimination depended on the illuminations and geometries and that the ability to discriminate the spatial consistency of specular highlights in glossiness perception showed larger individual differences than in other tasks. In addition, analysis of visual features showed that the parameters of higher order color texture statistics can partially, but not completely, explain task performance. The results obtained through crowdsourcing were highly correlated with those obtained in the laboratory, suggesting that our database can be used even when the experimental conditions are not strictly controlled in the laboratory. Several projects using our dataset are underway.